1. Field of the Invention
This invention relates generally to computer testing, and more particularly to automated test summary report generation.
2. Description of the Related Art
Business and other application programs execute in a wide variety of operating environments. Before deploying an application program, it is often desirable to test the application program to determine its suitability for the operating environment and to discover any problems associated with the application program before any undesirable physical, economic, or other damage may occur.
As the number and complexity of application programs and operating environments continue to increase, along with the potential damage that defects and problems associated with application programs may cause in some circumstances, techniques for testing application programs have become increasingly important. In conventional application program testing, a simulation of the operating environment in which the application program is intended to execute is generated. The application program is then executed within the simulated operating environment and the test results obtained.
FIG. 1 is diagram showing a prior art testing cycle 100. During a conventional test cycle, a test application 102 is executed on a particular platform 104. Generally, the test application 102 includes a plurality of test suites, which are used to test various aspects of an environment. For example, the test application 102 can include a plurality of test suites to test a particular Java application program interface (API).
The results of the test execution are then captured in a test execution log file 106. The test execution log file 106 includes detailed descriptions of which tests were executed and the results of each test. Generally, the test execution log file 106 is a long document having very detailed information. As such, the test execution log file 106 can be difficult to interpret. Thus, the testing team running the tests often manually analyzes the test execution log file 106 to determine which tests pass and which tests fail. In the case of failures, the testing team uses the test execution log file to determine where the failures are occurring and why the failures are occurring. These results are then summarized in a test summary report 108.
As the name implies, the test summary report 108 provides a summary of the detailed testing information contained in the test execution log file 106. In addition, the test summary report 108 includes a summary of the analysis performed on the test execution log file 106. Often, the test summary file 108 lists the tests that were executed and whether the test passed or failed. For example, the test summary report of FIG. 1 illustrates two test suites, test suite X and test suite Y. Test suite X has thirty tests of which twenty-eight passed and two failed. Test suite Y has forty tests of which fourteen passed and twenty-six failed. Thus, to create the test summary 108 of FIG. 1, the testing team must manually analyze the test results in the test execution log file 106 for seventy tests.
The test summary report 108 is then distributed to the appropriate personal, such as the project manager or development team. Although the conventional testing cycle 100 described above can provide test summary reports 108, the conventional process of generating the test summary report 108 is very time intensive and subject to errors. For example, manual counting of the tests pass and fail results is subject to counting errors, in which case the entire test suite would need to be reexamined. Moreover, conventionally over twenty hours per platform 104 is often needed to produce the test summary report 108.
In view of the foregoing, there is a need for systems and method for test report generation. The methods should provide test summary reports in an automated manner to reduce total test cycle time requirements and human error.